Repository navigation
Index ONE_OF and NOT_ONE_OF lists when the configuration loads - #13486
gh-worker-dd-mergequeue-cf854d[bot] merged 17 commits into
Conversation
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
The scale benchmark sat on the JSON value change and checked JSON integers by type, so it could not set a baseline on master. It now compares JSON numbers by value, covers every case file of the scale and scale-json suites, can isolate one case with the index parameter, and prints each case's index. A load benchmark measures parsing each suite's configuration and prints the heap it retains, so changes that move work to load time show their cost. The submodule points to the scale-suite fixture branch, which adds the N x M dimensions and JSON values up to 10 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each ONE_OF or NOT_ONE_OF evaluation scanned its list and compared the attribute with each element. With a number attribute, the scan also parsed each element as a double. A 10,000-element list cost about 25 us for a string attribute and 90 us for a number attribute. The parser now builds a MembershipIndex for each list of 2 or more elements: a HashMap for string comparisons and primitive long-keyed tables for numeric comparisons, each mapping a key to the element's first position. A lookup costs about 2-4 ns at every list size. The index gives the same result as the scan, including the numeric leniency (42 matches "042") and the errors that the scan raises at an unconvertible element. When a result depends on such an error, or for an attribute type that the index does not handle, the evaluator runs the scan as before.
db2555d to
2999aae
Compare
…the index The long table hashed with the low bits of a Fibonacci product. A whole number below 2^21 has a key whose low 32 bits are zero, so small integers and powers of two all started at slot 0, and a lookup in a list such as 1..1000 walked about 500 slots (792 ns instead of 5 ns). The slot now takes the high bits of the product, which depend on every bit of the key. - MembershipIndexTest: start slots of small integers, powers of two, prices in cent steps and large IDs; it fails on the old hash. - UniversalFlagConfigParserTest: the parser builds the index for lists of two or more elements, and not for one element. - FlagEvaluatorMembershipTest: attribute types that the index leaves to the scan (Instant, list, map), and nested list and object elements. - A list without numeric strings shares one table for both number lookups. - Comments match the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scale benchmarks failed the forbidden API check on System.out, and nothing outside a manual JMH run checked the scale cases, so a sibling change that resolves a large configuration wrongly would go unnoticed. - Add ScaleSuiteTest: evaluate every case of the scale and scale-json suites twice, and fail when the benchmark misses a case file. The cases, their check and the suite reader move to the test sources and the benchmarks reuse them; the reader uses JsonReading. - Allow System.out in the benchmark setups (@SuppressForbidden). - Check every case again after the measurement, fail on an error code, and reject expectations the check does not handle. - Rerun the jmh task when the fixtures change. - Measure retained heap after a first parse, as the smallest of three signed samples. - Describe a row as the unweighted mean over the cases of a file, say that hooks are not measured and when to run three forks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture PR now checks that the committed scale files match the generator and fixes its docs. The case and configuration files are unchanged, so the benchmark numbers do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures add a case file with ONE_OF lists of the integers 1 to 1000 and of the powers of two up to 2^52, which an index keyed on exact doubles must still match. Pin them and benchmark the file; its 8 cases pass on master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OpenFeature client passes the provider a new LayeredEvaluationContext on each call, while the benchmark reused one MutableContext. A rule on `id` when the context has no `id` reads the layered context's key set, which it builds again on each call, so the benchmark did not show what applications pay for it. evaluateLayered wraps each case's context the way the client does, as ExperimentationOptOutBenchmark.evaluateFreshLayeredContext does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a114d9fc59
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
More details
The index preserves numeric coercion and list-order error behavior, with scan fallback for compound comparisons. Tests and performance benchmarks were not rerun.
🤖 Bits Code Review · Commit a114d9f
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
leo.romanovsky@datadoghq.com cancelled this merge request build |
|
/merge -c |
|
View all feedbacks in Devflow UI.
|
The HashMap of strings to first positions held a node and often a boxed Integer per value. On the scale configuration it added 4.4 MB in about 130,000 long-lived objects that every concurrent mark traces. A flat open-addressed table, modeled on StringIndex, keeps the strings, their cached hashes and, only for lists with a number element, their first positions. Lists without strings or without numbers share one empty table. Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/merge |
|
View all feedbacks in Devflow UI.
It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.
The expected merge time in
|
ac1d3cf
into
master
What Does This Do
ONE_OFandNOT_ONE_OFconditions use a hash index that the parser builds when the configuration loads. A membership test costs about 2–4 ns at every list size. Before this change, each evaluation scanned the whole list.The results do not change. This includes the numeric leniency (
42matches"042") and the errors that the scan raises at an element that it cannot convert to a number.Motivation
Customers target large lists of accounts, users, and devices with
ONE_OF. The scan cost grows with the list size, and with a number attribute it also parses each element as adoubleon each evaluation:Changes
ufc/MembershipIndex(new class):strings: a flat open-addressing table ofString.valueOf(element)for the elements that are not numbers, modeled onStringIndexininternal-api(which this module cannot depend on):String[]slots, cachedint[]hashes that gateequals, at most half full. It keeps each string's first position only when the list has a number element, as only then does the position decide a match.numericandnumbers: open-addressing tables with primitivelongkeys (the bits of thedoublevalue, with-0.0stored as0.0). They map each element that converts to a number, and each number element, to its first position. A lookup does not box. The slot is the high bits of a Fibonacci product, so whole numbers and powers of two spread over the table. A list without numeric strings uses one table for both, and a list without numbers or without strings shares one empty table.lookup(attribute)returnsMATCH,NO_MATCH, the element where the scan throws, orSCAN. A number attribute of any type is looked up by value, and any other attribute by its string.UniversalFlagConfigParser: builds the index for eachONE_OFandNOT_ONE_OFlist of 2 or more elements.ConditionConfiguration: new transient fieldmembership.FlagEvaluator.isOneOf: uses the index. When the scan throws, it compares the attribute with the element where the scan throws, which throws the same exception. It runs the scan (scanOneOf, the code from before) when the index returnsSCANor when no index exists.MembershipIndexTest(new): bounds the probe lengths of hits and misses in the number table for integers, negative integers, powers of two, prices in cent steps and ending in.99, large IDs, and integers mixed with powers of two in one list.MembershipIndexStringTableTest(new): a collision chain that wraps past the end of the table, misses that collide with every key, the two strings whose hash code is 0, repeated keys, lists without numbers or without strings, mixed lists in list order, and the empty list.UniversalFlagConfigParserTest: the parser builds the index for lists of 2 or more elements, and not for 1 element.FlagEvaluatorMembershipTest(new):null, nested lists and maps,"-0","NaN","Infinity"," 42 ", and"4.2e1". The attributes are strings, integers, longs (one above 2^53), doubles (-0.0,NaN), booleans, anInstant, a list, and maps, one of them equal to a nested map that prints its keys in another order.Results
Membership structures alone, JMH AverageTime, ns per lookup (half of the keys are present):
TreeSetHashMapTreeMaplong[]This table measured a
HashMapfor strings. The string index is now a flat table, which the end-to-end runs below measure against theHashMap.Scale suite results
Same machine and settings as the scale-suite baseline of #13485: AWS workspace, Intel Xeon Platinum 8175M, 16 vCPUs, JDK 21.0.10, JMH AverageTime, 1 fork, 3 warm-up and 5 measurement iterations of 2 s,
-prof gc. Each row is the mean over the cases of the file. The runs were back to back.masterscale/one-of(10,000-entry lists)scale/one-of-many(500 × 100 UUIDs, 1,000 × 256 chars, 50 × 200)scale/one-of-small-numbers("1".."1000"and 53 powers of two)scale/rules,rules-sweep,time-windows,variations,flagsA smoke run against
masterat 4075f1f (same machine, JDK and JMH settings, back to back, load average 3 to 13) gives:one-of115 µs and 80,421 B onmasteragainst 372 ns ± 157 and 374 B on this PR;one-of-many156 µs against 11.5 µs ± 4.4;one-of-small-numbers11.5 µs and 15,424 B against 316 ns ± 76 and 380 B. The other case files stayed within the error. This is a shared workspace, so the errors on this PR's rows are wide; the ratios match the table.one-of-manystill costs 11.6 µs, because 500 allocations each do one lookup. The remaining cost is the allocation scan, not the lists.The cost at load is in the next section.
Retained memory
The parsed configuration lives as long as the process, and every concurrent mark traces it, so the object count matters as much as the bytes. JOL
GraphLayout.parseInstanceon the parsedscaleconfiguration (10,030 flags; 576 indexed lists of 92,093 values), JDK 21.0.10:masterat 4075f1fHashMapAllocation and parse time come from
ScaleSuiteLoadBenchmarkwithsuite=scaleand-prof gc, 3 forks of 5 × 2 s, at a load average of 6 to 8; the parse times are within the error of each other. The added allocation is the tables and the build arrays of each list, which the parse drops.The
HashMapheld a node, and often a boxedInteger, for each value: 4.4 MB of the 5.05 MB. The flat table needs about 4 objects for each list. On the canonical fixtureufc-config.json, the index owns 20 objects instead of 43.The flat table keeps the speed.
ScaleSuiteBenchmarkone-of rows, the two jars alternating, 6 forks of 5 × 2 s each,-prof gc, medians:masterHashMapscale/one-ofscale/one-of-manyscale/one-of-small-numbersBoth allocate the same 374, 368, and 380 bytes per operation;
masterallocates 80,422, 393, and 15,424. Other sessions shared the workspace, with a load average between 2 and 9.Decisions
TreeSetand 10 times faster than a binary search at 10,000 elements. A JVM server can pay the build cost when the configuration loads.42would no longer match"042"), and the errors would be lost. The index stores the first position of each key and of the first element that cannot convert to a number, so it can tell which result the scan gives. When the scan's result is an error, the evaluator compares the attribute with the element where the scan throws, so the error type and message stay the same without a scan.HashMap<Double, Integer>boxes each lookup. Thelong-keyed table does not allocate, and it is sized only for the elements that convert to numbers. A list of non-numeric IDs gets an almost empty number table.key * 0x9E3779B97F4A7C15L. For whole numbers below 2^21 and for powers of two, the low 32 bits of the key are zero, so the low bits of the product are too. A slot taken from them puts all of1..1000at one slot: 500 probes for each hit inMembershipIndexTest, against 1.06 with the high bits.ONE_OFcondition over a 2-element string list costs 178 ns when the attribute is first (scan: 178 ns), 181 ns when it is last (scan: 198 ns), and 101 ns when it is absent (scan: 108 ns). At 16 elements, the last element costs 191 ns against 333 ns. A one-element list keeps the scan.HashMap. TheHashMapcost 60–80 bytes and one or two objects for each string element. The flat table costs 8 bytes for each slot (a reference and a hash), plus 4 for a position when the list has a number element, in 2 to 3 arrays for each list. Number elements cost 13 bytes for each slot. The configuration already holds the list and its strings, so the table only refers to them.Additional Notes
Limitations:
Listor aMap, against a list that has a nested list or object. Two equal maps can print their keys in a different order, so the string does not decide. This case costs O(n), as before.42matches"042"in Java, as before.Follow-ups:
StringIndex. The string table can becomeStringIndexonce the toolbox is shaded into this module (Shade the perf toolbox into dd-openfeature (FFE optimization) #13532).int[]hash table) could replace the lists themselves, taking the lists and their strings from about 8.5 MB to 4.7 MB onscale. The evaluator and the parser read the lists today, so this belongs in its own change.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issue